File storage system

1 Introduction and Goals

This document provides an architectural overview of the File Storage Service (FSS), a solution designed to store and manage files, such as images, videos, and other media, for external applications.

1.1 Requirements Overview

Features:

Upload, download, and delete files via RESTful APIs.
Metadata management for stored files.
Access control mechanisms to manage file visibility and permissions.
Versioning for stored files.
Storage files on local file system or in AWS S3.

2 Architecture Constraints

Constraint	Description
API Protocols	The service must use REST APIs
File Storage Mechanism	Files must be stored using an object storage solution on local files system or AWS S3

3 Context and Scope

4 Solution Strategy

Goal	Description
Security	OAuth2 for secure user/externalSystem authentication

5 Building Block View

Diagram from 3.1. section displaying all systems:

Block	Description
File Storage System	Main system responsible for storage files.
Identity and Access Management (IAM) system	System responsible for authentication and user management.
Person	Person can use REST Client to manually sent file.
External System	Other system which store files.

5.1 Level 1 - File Storage System

The system architecture consists of two main components File Storage Service and supporting infrastructure.

System or components	Description
File Storage Service	The service is implemented in Spring Framework
supporting infrastructure	The system uses MongoDB as the primary database. Additionally Apache Kafka is utilized as a messaging system

To model the domain, the DDD (Domain-Driven Design) approach was applied, complemented by Event Storming. More information: Event Storming

5.1.1 Level 2 - File Storage Service

The service breaks down in 3 components:

container - component is responsible for containers management;
object - component is responsible for objects management;
storage - component is responsible for storing files;

5.1.2 File Storage Service - REST API documentation

Comprehensive documentation for the REST API can be accessed via the following link.

5.1.3 File Storage Service - Security

The service functions as a Resource Server. It utilizes scopes to determine whether the user has sufficient permissions to perform a specific action. The table below presents the basic roles required to obtain the corresponding scopes. For example, to obtain the scope fss.containers:read, the fss.containers-read role must be assigned.

Base role	Scope	Claims	Access Granted
fss.containers-read	fss.containers:read	-	Permission to read any container.
fss.containers-write	fss.containers:write	-	Permission to create, change or delete container.
fss.objects-read	fss.objects:read	-	Permission to read any object in container(PUBLIC or PRIVATE) and owns objects in PRIVATE containers.
fss.objects-write	fss.objects:write	-	Permission to create, change or delete object in container(PUBLIC or PRIVATE) and owns objects in PRIVATE containers.
fss.objects-manage	fss.objects:manage	-	Permission to read, create, change or delete any object in container.

Managing such basic roles individually would be cumbersome; therefore, two additional roles have been created to aggregate the basic roles. Role Mapping Matrix:

Role\Base role	fss.containers-read	fss.containers-write	fss.objects-read	fss.objects-write	fss.objects-manage
fss.admin	+	+	+	+	+
fss.user	-	-	+	+	-

Example decoded token for user with fss.admin role:

{
  "exp": 1735338196,
  "iat": 1735337896,
  "auth_time": 1735336999,
  "jti": "4509bfe4-9436-4874-8a74-de511e9c0232",
  "iss": "http://localhost:8080/realms/f18",
  "aud": "account",
  "sub": "9bb82d81-f938-43ef-8aa8-c84706deb6db",
  "typ": "Bearer",
  "azp": "fss",
//....
  "scope": "openid fss.containers:write profile fss.objects:read fss.objects:write email fss.objects:manage fss.containers:read",
  "email_verified": true,
  "name": "Ala Kot",
  "preferred_username": "Ala Kot",
  "given_name": "Ala",
  "family_name": "Kot",
  "email": "ala@ma.kota"
}

The table below describe the permissions required to perform actions on objects in container.

method \ container type	PUBLIC	PROTECTED	PRIVATE
fetch one object		fss.objects-read	owner or fss.objects-manage
fetch all objects	fss.objects-read	fss.objects-read	fss.objects-manage
edit object	fss.objects-write	fss.objects-write	owner or fss.objects-manage

5.1.4 File Storage Service - Container Components

Implemented in Layered architectural patter as Active Record.

The component enables:

creation,
modification, and
deletion of containers.

Each container is:

identified by a unique name
access types - define access of objects in container:
- Public: All objects within the container are accessible to anyone without requiring authentication.
- Protected: Objects are accessible only to authenticated actors.
- Private: Objects are restricted to the object owner or actor with specific authority.
revision Policy - define revision policy for objects in container:
- REVISION_ON - revision enable
- REVISION_OFF - revision disable

Example container:

{
"containerId": "676f243e808bd90f188658a8",
"name": "custom_container_name",
"type": "PUBLIC",
"revisionPolicyType": "REVISION_ON"
}

5.1.5 File Storage Service - Object Components

Implemented in Port & Adapters architectural patter as Domain Model.

The component enables:

creation of objects,
modification of objects,
modification of objects by restoring previous object's revision if revision enable,
deletion of objects,
deletion of object's revisions if revision enable.

Example object when revision Policy is enabled(revision 3 was deleted):

{
  "objectId": "676f2666808bd90f188658a9",
  "key": "god.jpg",
  "containerId": "676f243e808bd90f188658a8",
  "revisions": [
    {
      "revisionId": 1,
      "revisionType": "CREATED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_1.jpg"
    },
    {
      "revisionId": 2,
      "revisionType": "UPDATED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_2.jpg"
    },
    {
      "revisionId": 4,
      "revisionType": "DELETED",
      "size": 0,
      "storageId": null
    },
    {
      "revisionId": 5,
      "revisionType": "RESTORED",
      "size": 621697,
      "storageId": "676f2666808bd90f188658a9_5.jpg"
    }
  ],
  "metadata": {
    "extension": "jpg",
    "ownerId": "9bb82d81-f938-43ef-8aa8-c84706deb6db"
  }
}

5.1.6 File Storage Service - Storage Components

Implemented in Layered architectural patter as Transaction Script.

The component is responsible for file storage in Local File System.

5.2 Level 1 - Identity and Access Management (IAM) system

The system integrates Keycloak as the Identity and Access Management (IAM) solution, eliminating the need to develop custom authentication and authorization mechanisms. This choice reduces development effort, enhances security, and ensures compliance with modern authentication standards like OAuth2 and OpenID Connect.

6 Runtime View

// TODO

7 Deployment View

7.1 DEV

Deployment diagram - dev:

7.1.1 IAM

Run keycloak and postgresDB:

 docker-compose -p auth -f .\.docker\docker-compose-auth.yaml up -d

Config realm
1. Create realm using file realm-export.json
2. Create user
3. Regenerate Client Secret in fss Client

7.1.2 File Service

Run

Setup dev env:

 docker-compose -p f18 -f .\.docker\docker-compose.yaml up -d

Build everything:
```
.\gradlew service:file-storage:build
```

Start application:

 .\gradlew service:file-storage:bootRun

Test

Execute tests:
```
 .\gradlew service:file-storage:test
```

Docker

Publish image

Build image:

 docker build . -f .\.docker\file_storage_service_image\Dockerfile -t ghcr.io/grubeb/file-storage-service:latest --no-cache

Publish image:

  docker push ghcr.io/grubeb/file-storage-service:latest

8 Cross-cutting Concepts

8.1 Clean Architecture

Architecture is described in book: Get Your Hands Dirty on Clean Architecture, by Tom Hombergs.

9 Architecture Decisions

Link

10 Quality Requirements

11 Risks and Technical Debts

12 Glossary

Term	Definition

13 attachments

13.1 System modeling

13.1.1 introduction

To model the domain, the DDD (Domain-Driven Design) approach was applied, complemented by Event Storming. Event Storming is an excellent tool for supporting DDD processes, particularly during the exploratory and design phases. It facilitates the identification of domain events, the modeling of business processes, and the development of a shared language among all stakeholders.

Event Storming is created using draw.io, and the corresponding file is located in the designated folder(./f18.drawio). Below, an exported image of the documentation is provided for reference:

13.1.2 Domain Description

The File Storage System is designed to manage and store files along with their associated metadata. It consists of two primary entities: Containers and the Objects they contain.

A Container is a logical grouping of Objects and serves as the primary organizational unit in the system. Each container is:

identified by a unique name
is assigned one of the following access types:
- Public: All files within the container are accessible to anyone without requiring authentication.
- Protected: Files are accessible only to authenticated actors.
- Private: Files are restricted to the container owner or users with specific authority.
revision Policy:
- REVISION_ON - revision enable
- REVISION_OFF - revision disable

An Object represents a file stored in the system. Object consists of the following attributes:

Key: Derived from the file name,
Metadata: Includes additional information about the file, such as size, creation date, and custom attributes.

13.1.3 Strategic Design (ES: Pig Picture & Process Modeling)

During modeling, the following definitions were established as universal concepts, applicable across all bounded contexts within the domain:

container - logical container for objects;
object - represents a file stored in the system;

13.1.4 Legend:

img.png

13.1.5 ES. Pig Picture:

img.png

13.1.6 ES. Process Modeling:

container related processes
img.png
object related processes:
img_1.png

img_2.png
storage related processes:
img_3.png

13.1.7 Tactical Design (ES: Design Modeling)

img_4.png

After documenting and describing the processes, the following bounded contexts were identified:

1. container context

Implemented in Layered architectural patter as Active Record.

2. object context

Implemented in Port & Adapters architectural patter as Domain Model.

3. storage context

Implemented in Layered architectural patter as Transaction Script.